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INTRODUCTION 


Before considering thermal imaging itself, we must first examine the elements of a 
pattern recognition system. The function of a pattern recognition system is to make a 
decision concerning the class of membership of the patterns with which it is confron- 
ted. A number of information translation processes take place between the time that a 
pattern is input and a decision is made by the system. These processes are sum- 
marized in figure 1. The functions of the blocks shown in this figure are described 
briefly as follows: 


* The sensor is the measurement device that transforms the input 
patterns into a form suitable for machine manipulation. The preprocessor removes 
unnecessary elements of the measured data. The feature extractor then computes 
from preprocessed data the features required for classification. Finally, these features 
go into the classifier, which yields a decision concerning the class membership of the 
pattern being processed. 


¢ Although the general form of these operations is specified by the 
system designer, each specific operation is characterized by variable parameters that 
must be adopted to a given pattern recognition problem. The adjustment of these 
parameters is carried out using sample patterns in what is called a learning process. 


* Machining techniques are either of the supervised or unsupervised 
types. In the supervised type, the system parameters are estimated by algorithms that 
use sample patterns whose class membership is specified by the designer. The 
unsupervised learning approach is used when there is little prior knowledge about the 
pattern classes of a given problem. 


TWO-DIMENSIONAL TRANSFORMS ON IMAGES 


Let us now consider the mathematical transforms of images for pattern recog- 
nition. Let f(x,y) be the brightness of intensity samples of a discretized image and 
F(u,v) be the transform domain of an image. The Hadamard transform of an image can 
then be given as 


mes S gxyy(-1),55 UiXi + Viyi 
N xwO y=0 (1) 


where 
N = resolution of the digitized image 


u, V, X, and y = ith binary variables in the expansion of the variables u, v, x, and y, 
respectively. 


This transformation is unique in the sense that a second Hadamard transform on the 
function F (u,v) will result in the original image f(x,y). In addition, the one-dimensional 
version of the transform is obtained by N log N additions, if implemented on a digital 
computer. 


Using the same notation as before, the discrete Fourier transform becomes 


N-1 N-1 : 
Fuvy=1 > > t(xy)exp (2a (ux + w) 
N x=zOy=0 N (2) 
This transform is unique in the sense that a second Fourier transform on the function 
F;(u,v) will result in a rotation of the original image, f (-x,-y). The computer implemen- 
tation of the one-dimensional transform is obtained with an algorithm which requires 
N logeN and N/2 (logo(N)-2) + 1 multiplications. 


Both the Hadamard and the Fourier domains can be analyzed for classes of 
identifying signatures by the decomposition of the image into two-dimensional 
Radamacher-Walsh or trigonometric orthonormal wave forms, respectively. 


The properties of symmetry, dynamic ranges, and entropy of equation 2 will now 
be considered. The function f(x,y) describes the intensity of samples of an image’s 
spatial domain. Since intensities are non-negative and real, f(x,y) must have the same 
properties. With this restriction on f(x,y), the Fourier domain becomes its own sym- 
metric conjugate 


F (u,v) = F* (-u,-v) (3) 


This symmetry property requires that only half of the frequency samples need be 
retained for complete reconstruction of the original function, f(x,y). 


Another property of the Fourier domain is its large dynamic range of possible 
values. If A is an upper band on f (x,y), then the range of allowable values at the origin 
is 0 to AN and the bound on all other frequency samples is +AN/2. While the dynamic 


range of possible values is extremely large, only a few points at a time can actually 
take on very large values. Parseval’s relationship states that the total energies in the 
spatial and frequency domain must be equal 


S F tey)= S » IF (u,v)? 


x20 y=0 unO vsO0 (4) 


Only a few spectral points, then, can be large because of the energy bounds. It is this 
redistribution of energy in the frequency domain that makes the energy matched filter 
relatively insensitive to those spatial frequencies with lower amplitudes which usually 
contain high frequency information. 


A final property of the Fourier domain is obtained from information theory. The 
entropy of the Fourier domain is equal to that of the spatial domain. This means that 
information is neither gained nor lost when comparing the spatial and Fourier domains 
of an image. Although the entropies of the two domains are the same, equal 
bandwidths are not necessarily easily obtained. The Fourier domain contains larger 
valued numbers which require an increase in bandwidth for storage purposes. 
Consequently, a quantization technique must be developed if it is desired to represent 
the Fourier domain with the same number of bits as the spatial domain. 


IMAGE EVALUATION 


If 6 (x,y) is the output of spatial filter h (x,y) with input f (x,y) 


0 (xy)={2Jat(Gn)n(x-% y-n) dv dn (5) 
= f (xy) *h «y) (6) 
and 
@ (u,v) = F (u,v) H (u,v) (7) 
where 


* = convolution symbol 


@ (u,v), F (u,v), and H (u,v) = Fourier transforms of 6 (x,y), f (x,y), and h (x,y), respec- 
tively. 





Spatial frequency filtering can be used to remove noise or enhance, recognize, or 
evaluate images. The matched filter does pattern recognition as well as correlation of 
output data. The true matched filter maximizes the signal-to-noise ratio in a detection 
process when the noise perturbation is considered to be additive. {f the signal to be 
detected is f (x,y) and the unknown input is 


g (x,y) = f (x,y) +n (xy) (8) 


then the matched filter response is given by 


F * (u,v) 
AU) ae) (9) 


where 
F * (u,v) = conjugate of the Fourier transform of f (x,y) 
N (u,v) = power spectrum of noise process n (x,y) 


A class of match filters can be described by the transfer function 


Hp (u,v) = (-1)° (u2 + v2) F * (u,v) 


Np (u,v) (10) 


where 


Np (u,v) = power spectrum of pth gradient operator on noise process n (x,y) 
p = 0, for traditional energy matched filter 

p = 1, for gradient matched filter 

p = 2, for Laplacian matched filter 

p =n, nth order gradient matched filter 


The output of a filter in the presence of noise sampled at the origin is 


(u2 + v2)? F * (u,v) G (u,v) du dv 


2 (0,0) =f {2-1) Ais 


Peak correlation occurs when the filter matches an image with itself in the absence of 
noise 


@p (0,0) = f f2(-1)° IF (u.v)4 (u2 + v2)? du dv (12) 


The ratio®p, (0,0) / 6) (0,0) can then be defined as a correlation parameter, and an 
image evaluation decision can be based upon it. 


IMAGE PROCESSING 


Many applications demand nothing more than the classification of the contents of 
any image presented to an automatic system. The classification may range from a 
single bit to many bits, such as counts, by type, of the number of objects of several 
categories observed in the image. 


Other applications of image processing require image enhancement, in order to 
make otherwise obscure objects stand out. Some of the ways in which this is done is 
by increasing contrast or changing the gray-scale range, removing certain artifacts, 
and modifying the spatial frequency content of the imagery. 


Jn still other applications, bandwidth compression is used in order to conserve on 
the precious commodity. As an example, consider a Mars communications system 
having a bandwidth of 10 bits/s. It would require one day to transmit a single 16 gray- 
level picture. In view of the high cost of this information, one would consider reducing 
the data to a one-out-of-ten code identifying the Martian scenery as being most like 
one of ten different terrain textures. The bandwidth compression ratio in this case, 
then, is 10:1. On the other hand many applications allow the removal of only true 
redundancy, reducing the compression ratio of 10:1 to 3:1. 


IMAGE PROCESSING IMPLEMENTATION 


Image Processing Classifications 


Although there is an enormous variety of ways in which image processing can be 
implemented, it appears that nearly all hardware forms can be classified into one of 
four categories, depending upon the degree of parallelism inherent in the calculations 
being implemented. All conceptual approaches can be reduced to an equivalent 
network form. Because the object can appear anywhere on the image, it is assumed 
that the classification network must be repeatedly applied geometrically relative to 





many sampled elements in the image. These are called the image points. Within the 
network, the operator functions make calculations based upon inputs from their 
relevant image siements. These inputs are called the operator points. The basic 
hardware imc!ementations and their process time, based on a 1 ps operation time, are 
shown in .able 1. Each of these implementations will now be considered in turn. 


Table 1. Image processing implementations 


Image point Operator point Time per frame 
Serial Serial 21 days 
Serial Parallel 1 hr 
Parallel Serial 2 sec 
Parallel Parallel 6 ms 


image Serial - Operator Serial 


The cathode ray tube (CRT) scanner and digital computer is the most versatile 
method for processing images. {t has the ability of being programmed for any clas- 
sification concept. In the case in which the operators represent spacial frequencies, 
the fast Fourier transform method can accelerate evaluation considerably. With the 
fast Fourier method, a typical transform process on a 256 by 256 element image takes 
about 5 min of computer time. Converting this figure to the same basis as table 1 
yields 15 hr instead of 21 days. Although this is a great improvement, it is far from 
acceptable. One of the following systems must be implemented to obtain a practical 
system. 


image Serial - Operator Parallel 


Various methods have been devised for generating the operator connections in 
parallel, and scanning over the image area serially. A typical technique employs an 
optical mask to mechanize the weighted sum of input values of the linear threshold 
function together with a scanned beam to serially mechanize the threshold. 


image Parallel - Operator Serial 


The effect of performing a single operation over an entire image can be ac- 
complished by sequentially shifting the image in amounts corresponding to the 
operator connections and at each position adding the image to itself. This type of 
system can be implemented by electron imaging devices, which provide temporary 


parallel integration of a sequence of exposures. Another implementation is with a 
parallel computer. In such a computer fields of data can be shifted in parallel, and 
simple logic processes can also be performed in parallel. By programming the 
sequence of logic operations the usual arithmetic functions, as well as any threshold 
operations, can be generated. 


Image Parallel - Operator Parallel 


A truly parallel system can be made with optical and photographic methods. The 
main disadvantage of the implementation is the long time required to process the film 
used for intermediate storage of processing results. The solution to paralle!-paralle! 
processing lies in the combination of parallel optics and parallel image amplification. 


THERMAL IMAGING 
introduction 


The molecules of a medium absorb infrared (IR) radiation at characteristic 
wavelengths. In the case of the atmosphere, the chief absorbing constituents in the IR 
band are carbon dioxide, water vapor, and ozone. The transmission curve for IR 
through the atmosphere is shown in figure 2. It is seen that there are two windows in 
the range of 3 to 5 ym and from 8 to 13 pm. For efficient transmission, the thermal 
system must operate in one of these windows. 


A block diagram of a thermal imaging system is shown in figure 3. The lens 
system works just like an optical lens system with the important difference that the 
lenses are made of a material that transmits infrared. The scanner and detector, which 
will be described more fully, are the heart of the system. The output of the detector is 
a video signal, and in many cases has the form of a standard television signal. The 
signal processing is, therefore, similar to television signal processing and the display 
appears on a television monitor. 


Scanning and Detecting 


The ideal detector consists of a mosaic of elements resembling the eye and not 
requiring scanning, since the whole scene appears on the array. A resolution of 0.5 
mrad and a field of view of 20 deg requires approximately half a miliion picture points. 
Each detector must have its own preamplifier and each channel must have an 
identical response. 


Going to the other extreme, one could have a single element that can be scanned 
across the whole scene (fig. 4). For television picture quality, an information picture 


bandwidth of about 5 MHz is required. The cadmium mercury telluride photoconductor 
has one of the fastest response times, but its bandwidth of around 200 kHz is clearly 
inadequate. 


An array of detector elements, therefore, must be used to get both high thermal 
sensitivity and picture quality. The use of the detector array with N elements improves 
the signai-to-noise ratio by the yyy. This is because the sampling rate for any one 
picture is increased by a factor of N; the signal is increased by N times and the 
incoherent noise by the yyy. 


The layout of a parallel array is shown in figure 5. It is mounted vertically and 
scanned horizontally. The whole scene may be scanned in one sweep provided the 
array contains enough elements for adequate resolution. The same result can be 
obtained with a series of continuous swathes from top to bottom (fig. 6). The output of 
each detector is amplified and displayed sequentially on a CRT or light emitting diode 
(LED) display. A typical detector system of 150 elements; each 50 ym square, a frame 
rate of 25/s, a 300 line picture, and suitable optics; would have a sensitivity of 0.5 C. 


Thermal Detectivity 
The simplest method of forming a thermal image using photon detectors is to 


scan all points of an image of the scene and in turn with this modulate the brightness 
of the corresponding display. Using a scanning system has the advantage of ac 


coupling; only the signal variations are passed, while the large dc pedestal is rejected. 
An artificial dc 'evel is added by an internal reference and the available contrast is 
expanded over the dynamic range of the display. The contrast expansion requires a 
good signal-to-noise ratio; detector noise sets a lower limit to the smallest detectable 
temperature difference. 


An equation for the temperature sensitivity of an image can be derived from the 
detector response, the collecting power and efficiency of the optics, and the scanning 
system. We proceed as follows: 

NEP, = AH), (13) 
where 


NEP = noise equivalent power, w 


A = detector area, cm2 


H = incident flux at wavelength A for unit S/N, w/cm2 





Because using NEP has some inconvenience, the quantity D* was adapted to 
describe detector performance, which includes NEP and also normalizes for the 
inverse square root dependence of the photon detector on area and bandwidth. 





Thus 
1/2 
p,* = AB)". Bt? 
NEP, A'?H, (14) 
where 
A = area, cm2 


B = bandwidth, Hz 
D,* = detectivity, Hz1/2w-icm 


The bandwidth for a system employing a single element detector is 


1 S¢f 
Bao nt (15) 


where 
€ = scanning efficiency 
6 = vertical field of view, rad 
6 = horizontal field of view, rad 
f = frame frequency, Hz 
m = angular resolution, rad2 


lf a detector array with N elements is used in parallel scan, the bandwidth per channel 
becomes B/N. 


For a temperature difference in a scene to be just detectable by a scanning 
imager, H, becomes the flux change at the detector at which signal strength equals 


noise. This is given by 


bis (aW/eT), AT ta to 
4F? 


where 


F = focal length/diameter 
ta = atmospheric transmission at wavelength 1 
to = optics transmission factor 


AT = temperature difference in scene to give Ha 
(QW/dT), = differential radiance from a black body at scene temperature 
Substituting Hy in equation 14 


4F°B 2 


D,= 
*” A'2 (awa), AT ta to 


FA 1/2 
AT= - 
A'to (WoT), ta D, (18) 


AT was derived for the wavelength A. If the effect of all wavelengths is included, the 
minimum detectable temperature difference becomes 


NEAT=——___4F°8 "7 


A‘ to (2) taD, dv 
op NOT Ja 





We now define M* as a figure of merit for detectors used in thermal imaging 


w | (2 taD* da 
then 
2n1/2 
NEAT = 46-8 
A‘? to M* (21) 


From equation 21 it is apparent that the greatest gain in sensitivity can be 
achieved by minimizing F, but the need to maintain resolution at the edges of the field- 
of-view limits this. Temperature sensitivities, NEAT, around 0.1 C are typical, but 
higher sensitivities to 0.001 C can be achieved for sp¢r’al purposes. 


RESULTS 


General Description 


A reali time thermal imaging system has an integral display using LED in a 
geometrical configuration corresponding to the detector array. The LED are driven by 
the corresponding detector outputs and the image is created by scanning the LED 
from the back of an IR mirror. The more common type of system has electronic 
processing that converts the amplified detector outputs into a sequence and form for 
television type display. From the theory presented in this report, four basic types of 
scan-imaging systems result. 


Serial Scan - Standard Video 

A serial scan--standard video system is shown in figure 7. It is the most direct 
way for generating standard video and has the advantage of not requiring multiplexers 
and scan converters. Another advantage is the low cost for displays, video recorders, 
symbol generators, data retransmitters, and automatic target rackers. 
Serial Scan - Parallel Video 


A basic serial scan--parallel video system is shown in figure 8. A two- 
dimensional array of detectors is coupled one-for-one to a similar array of LED. In this 
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system, the inherent integration of the observer's eye. produces the image-smoothing 
effect associated with serial scanning and combines this with the simplicity of a directly 
viewed LED array. 


Parallel Scan - Standard Video 


A parallel scan--standard video system is shown in figure 9. This stem requires 
a collecting converging lens, a two-sided scanner, an LED array, a vi _and a CRT. 
The video is converted to standard video format by electro-optical m. = »lexing with a 
television camera. For proper operation, perfect synchronization of the frame rates of 
the thermal and television scans is necessary, and matching of the dynamic ranges of 
the two devices is delicate. 


Parallel Scan - Parallel Video 


The simplest multidetector system is the parallel scan--parallel video system (fig. 
10). This system requires only a collecting converging lens, a two-sided scanner, a 
detector array, a set of amplifiers, display drivers, and an eyepiece. In the system, a 
convergent beam scanner uses one side of an oscillating thin mirror to generate the 
thermal scan and the other side to generate the visible scan. The detector signals are 
amplified and shaped to appropriately drive the visible light emitters. The mirror and 
eyepiece magnify and focus the LED scan for the observer. 


The advantages of this system are that no scanner synchronizer is needed, the 
scanner requires little power, and the display is compact. Its disadvantages are that 
only one person may observe the display, loss of nay part of channel causes loss of 
one line of video, all channels must be balanced individually and controlled simul- 
taneously, and video waveform shaping must be performed in each channel. Despite 
its disadvantages, the parallel - parallel system is a very cost effective way to achieve 
a small viewer. 
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Figure 1. Pattern recognition system 
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Figure 3. Thermal imaging system 
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Figure 4. Single detector serial scanning 
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Figure 6. Segmental parallel scanning 
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Figure 7. Serial scan - standard video system 
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Figure 8. Serial scan - parallel video system 
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Figure 9. Parallel scan - standard video system 
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Figure 10. Parallel scan - parallel video system 
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